{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "显卡日志\n",
    "\n",
    "下面给出了3090显卡的性能测评日志结果，每一条日志有如下结构：\n",
    "```\n",
    "Benchmarking #2# #4# precision type #1#\n",
    "#1#  model average #2# time :  #3# ms\n",
    "```\n",
    "其中#1#代表的是模型名称，#2#的值为train(ing)或inference，表示训练状态或推断状态，#3#表示耗时，#4#表示精度，其中包含了float, half, double三种类型，下面是一个具体的例子：\n",
    "\n",
    "```\n",
    "Benchmarking Inference float precision type resnet50\n",
    "resnet50  model average inference time :  13.426570892333984 ms\n",
    "```\n",
    "请把日志结果进行整理，变换成如下状态，model_i用相应模型名称填充，按照字母顺序排序，数值保留三位小数：\n",
    "\n",
    "```\n",
    "Train_half\tTrain_float\tTrain_double\tInference_half\tInference_float\tInference_double\n",
    "model_1\t0.954\t0.901\t0.357\t0.281\t0.978\t1.130\n",
    "model_2\t0.360\t0.794\t0.011\t1.083\t1.137\t0.394\n",
    "…\t…\t…\t…\t…\t…\t…\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "思路：\n",
    "\n",
    "首先查看txt文件，发现开头和结尾的一些部分是没用的，先去掉；\n",
    "\n",
    "之后再用正则表达式筛选出模型名，训练预测状态，精度与时间\n",
    "\n",
    "之后还有长宽表的转换"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11</td>\n",
       "      <td>mnasnet0_5  model average train time :  28.527...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12</td>\n",
       "      <td>Benchmarking Training float precision type mna...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>mnasnet0_75  model average train time :  34.10...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>14</td>\n",
       "      <td>Benchmarking Training float precision type mna...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15</td>\n",
       "      <td>mnasnet1_0  model average train time :  34.313...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index                                                  0\n",
       "0     11  mnasnet0_5  model average train time :  28.527...\n",
       "1     12  Benchmarking Training float precision type mna...\n",
       "2     13  mnasnet0_75  model average train time :  34.10...\n",
       "3     14  Benchmarking Training float precision type mna...\n",
       "4     15  mnasnet1_0  model average train time :  34.313..."
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df1 = pd.read_table('data/benchmark.txt',header=None)\n",
    "df1.drop([0,1,2,3,4,5,6,7,8,9,10],inplace=True)\n",
    "df1.reset_index(inplace=True)\n",
    "df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type_x</th>\n",
       "      <th>precision</th>\n",
       "      <th>model_name_x</th>\n",
       "      <th>model_name_y</th>\n",
       "      <th>type_y</th>\n",
       "      <th>time_val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Training</td>\n",
       "      <td>float</td>\n",
       "      <td>mnasnet0_75</td>\n",
       "      <td>mnasnet0_5</td>\n",
       "      <td>train</td>\n",
       "      <td>28.527636528015137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Training</td>\n",
       "      <td>float</td>\n",
       "      <td>mnasnet1_0</td>\n",
       "      <td>mnasnet0_75</td>\n",
       "      <td>train</td>\n",
       "      <td>34.10548686981201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Training</td>\n",
       "      <td>float</td>\n",
       "      <td>mnasnet1_3</td>\n",
       "      <td>mnasnet1_0</td>\n",
       "      <td>train</td>\n",
       "      <td>34.31377410888672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Training</td>\n",
       "      <td>float</td>\n",
       "      <td>resnet18</td>\n",
       "      <td>mnasnet1_3</td>\n",
       "      <td>train</td>\n",
       "      <td>35.556888580322266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Training</td>\n",
       "      <td>float</td>\n",
       "      <td>resnet34</td>\n",
       "      <td>resnet18</td>\n",
       "      <td>train</td>\n",
       "      <td>18.660082817077637</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     type_x precision model_name_x model_name_y type_y            time_val\n",
       "0  Training     float  mnasnet0_75   mnasnet0_5  train  28.527636528015137\n",
       "1  Training     float   mnasnet1_0  mnasnet0_75  train   34.10548686981201\n",
       "2  Training     float   mnasnet1_3   mnasnet1_0  train   34.31377410888672\n",
       "3  Training     float     resnet18   mnasnet1_3  train  35.556888580322266\n",
       "4  Training     float     resnet34     resnet18  train  18.660082817077637"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type1 = 'Benchmarking (?P<type_x>\\w+) (?P<precision>\\w+) precision type (?P<model_name_x>\\w+)'\n",
    "type2 = '(?P<model_name_y>\\w+)  model average (?P<type_y>\\w+) time :  (?P<time_val>.+) ms'\n",
    "\n",
    "info1 = df1[0].str.extract(pat1).dropna().reset_index(drop=True)\n",
    "info2 = df[0].str.extract(pat2).dropna().reset_index(drop=True)\n",
    "info3 = pd.concat([info1,info2],axis=1).reset_index(drop=True)\n",
    "info3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "info4 = info3.rename(columns={'model_name_x':'model','type_y':'type'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info4.pivot(index=['model','type'], columns='precision', values='time_val').dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "水压站点的特征工程\n",
    "\n",
    "df1和df2中分别给出了18年和19年各个站点的数据，其中列中的H0至H23分别代表当天0点至23点；df3中记录了18-19年的每日该地区的天气情况，请完成如下的任务：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df1 = pd.read_csv('data/yali18.csv')\n",
    "df2 = pd.read_csv('data/yali19.csv')\n",
    "df3 = pd.read_csv('data/qx1819.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过df1和df2构造df，把时间设为索引，第一列为站点编号，第二列为对应时刻的压力大小，排列方式如下（压力数值请用正确的值替换）：\n",
    "```\n",
    "                         站点    压力\n",
    "2018-01-01 00:00:00       1    1.0\n",
    "2018-01-01 00:00:00       2    1.0\n",
    "...                     ...    ...\n",
    "2018-01-01 00:00:00      30    1.0\n",
    "2018-01-01 01:00:00       1    1.0\n",
    "2018-01-01 01:00:00       2    1.0\n",
    "...                     ...    ...\n",
    "2019-12-31 23:00:00      30    1.0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>MeasName</th>\n",
       "      <th>H</th>\n",
       "      <th>压力</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>站点4</td>\n",
       "      <td>H0</td>\n",
       "      <td>0.402750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>站点7</td>\n",
       "      <td>H0</td>\n",
       "      <td>0.214375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>站点22</td>\n",
       "      <td>H0</td>\n",
       "      <td>0.247000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>站点21</td>\n",
       "      <td>H0</td>\n",
       "      <td>0.284250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>站点20</td>\n",
       "      <td>H0</td>\n",
       "      <td>0.292875</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Time MeasName   H        压力\n",
       "0  2018-01-01      站点4  H0  0.402750\n",
       "1  2018-01-01      站点7  H0  0.214375\n",
       "2  2018-01-01     站点22  H0  0.247000\n",
       "3  2018-01-01     站点21  H0  0.284250\n",
       "4  2018-01-01     站点20  H0  0.292875"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4 = pd.concat([df1,df2],axis=0)\n",
    "\n",
    "df4 =df4.melt(id_vars =['Time','MeasName'],\n",
    "             value_vars =['H0','H1','H2','H3','H4','H5','H6','H7','H8','H9','H10','H11','H12','H13','H14','H15','H16','H17','H18','H19','H20','H21','H22','H23'],var_name = 'H',\n",
    "             value_name = '压力')\n",
    "df4.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df4.rename(columns={'MeasName':'站点'})\n",
    "df4.H=df4.H.str.replace('H','')\n",
    "df4.Time=df4.Time+'-'+df1.H\n",
    "df4.Time=pd.to_datetime(df4.Time,format='%Y-%m-%d-%H')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在上一问构造的df基础上，构造下面的特征序列或DataFrame，并把它们逐个拼接到df的右侧\n",
    "\n",
    "当天最高温、最低温和它们的温差\n",
    "当天是否有沙暴、是否有雾、是否有雨、是否有雪、是否为晴天\n",
    "选择一种合适的方法度量雨量/下雪量的大小（构造两个序列分别表示二者大小）\n",
    "限制只用4列，对风向进行0-1编码（只考虑风向，不考虑大小）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对df的水压一列构造如下时序特征：\n",
    "\n",
    "当前时刻该站点水压与本月的相同整点时间该站点水压均值的差，例如当前时刻为2018-05-20 17:00:00，那么对应需要减去的值为当前月所有17:00:00时间点水压值的均值\n",
    "当前时刻所在周的周末该站点水压均值与工作日水压均值之差\n",
    "当前时刻向前7日内，该站点水压的均值、标准差、0.95分位数、下雨天数与下雪天数的总和\n",
    "当前时刻向前7日内，该站点同一整点时间水压的均值、标准差、0.95分位数\n",
    "当前时刻所在日的该站点水压最高值与最低值出现时刻的时间差"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>日期</th>\n",
       "      <th>天气</th>\n",
       "      <th>气温</th>\n",
       "      <th>风向</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>多云</td>\n",
       "      <td>1C～-4C</td>\n",
       "      <td>东南风 微风</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-01-02</td>\n",
       "      <td>阴转多云</td>\n",
       "      <td>8C～0C</td>\n",
       "      <td>东北风 3-4级</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-01-03</td>\n",
       "      <td>阴转小雪</td>\n",
       "      <td>1C～-1C</td>\n",
       "      <td>东北风 4-5级转4-5级</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-01-04</td>\n",
       "      <td>阴</td>\n",
       "      <td>0C～-4C</td>\n",
       "      <td>东北风转北风 3-4级转3-4级</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-01-05</td>\n",
       "      <td>阴转多云</td>\n",
       "      <td>3C～-4C</td>\n",
       "      <td>西风转北风 3-4级转3-4级</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>2019-12-27</td>\n",
       "      <td>多云转晴</td>\n",
       "      <td>6℃～-1℃</td>\n",
       "      <td>西南风转南风 3-4级转3-4级</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>725</th>\n",
       "      <td>2019-12-28</td>\n",
       "      <td>多云转小雨</td>\n",
       "      <td>10℃～4℃</td>\n",
       "      <td>西南风转南风 3-4级转3-4级</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726</th>\n",
       "      <td>2019-12-29</td>\n",
       "      <td>多云</td>\n",
       "      <td>11℃～2℃</td>\n",
       "      <td>西南风转北风 &lt;3级转&lt;3级</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>727</th>\n",
       "      <td>2019-12-30</td>\n",
       "      <td>阴转晴</td>\n",
       "      <td>4℃～-6℃</td>\n",
       "      <td>东北风转北风 4-5级转4-5级</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>2019-12-31</td>\n",
       "      <td>晴转多云</td>\n",
       "      <td>0℃～-5℃</td>\n",
       "      <td>西风转南风 &lt;3级</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>729 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             日期     天气      气温                风向\n",
       "0    2018-01-01     多云  1C～-4C            东南风 微风\n",
       "1    2018-01-02   阴转多云   8C～0C          东北风 3-4级\n",
       "2    2018-01-03   阴转小雪  1C～-1C     东北风 4-5级转4-5级\n",
       "3    2018-01-04      阴  0C～-4C  东北风转北风 3-4级转3-4级\n",
       "4    2018-01-05   阴转多云  3C～-4C   西风转北风 3-4级转3-4级\n",
       "..          ...    ...     ...               ...\n",
       "724  2019-12-27   多云转晴  6℃～-1℃  西南风转南风 3-4级转3-4级\n",
       "725  2019-12-28  多云转小雨  10℃～4℃  西南风转南风 3-4级转3-4级\n",
       "726  2019-12-29     多云  11℃～2℃    西南风转北风 <3级转<3级\n",
       "727  2019-12-30    阴转晴  4℃～-6℃  东北风转北风 4-5级转4-5级\n",
       "728  2019-12-31   晴转多云  0℃～-5℃         西风转南风 <3级\n",
       "\n",
       "[729 rows x 4 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
